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Vocational education research and development (R&D) 
product evaluations are difficult to conduct for several reasons, 
such as the many types of R&D products, the multiple users of the 
products with different goals for evaluation, the limited time 
available, and the high cost. In order to meet these challenges, an 
internal evaluation team at the National Center for Research in 
Vocational Education developed a conceptual framework for evaluating 
R&D product impact. The conceptual framework was a linear model with 
five stages: development, distribution, implementation, utilization, 
and effects. Two different types of impact criteria were presented in 
the framework: formative and summative. A more specific model was 
created to measure product impact. The formative criteria that were 
addressed in this model were user orientation, support systems, and 
integrated utilization; and the summative criteria were user 
satisfaction and individual growth. A modeling approach known as 
structural equation modeling can be used to determine the degree to 
which the proposed model actually represents the R&D product impact 
phenomenon in the population. LISREL, an acronym for linear 
structural relationships, is a computer program developed to analyze 
structural equation models. The results of a LISREL analysis will 
give evidence of the fit of the model in the population and the fit 
of individual parameters. LISREL possesses a number of features that 
bring enormous potential for analyzing other vocational education R&D* 
phenomena. (References on LISREL are included in the report.) (KC) 
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AN APPLICATION OF MODELING TO 
VOCATIONAL EDUCATION R&D PRODUCT EVALUATION 

Debra DeVore Bragg and William L. Hull 

The evaluation of research and development (R&D) products has been 
a highly specialized endeavor conducted primarily by -federal agencies, 
■federally-sponsored research centers and some state departments of 
education. Some individuals who have conducted R&D product evaluations 
have found their evaluation responsibilities frustrating, yet usually 
challenging. The necessity for understanding the impact of R&D products 
has perpetuated evaluation efforts in spite of the complexity involved 
in designing and conducting impact evaluations. Even though the 
approaches to conducting impact evaluation of R&D products have been 
widely debated, diverse approaches have revealed benefits of R&D product 
use. Impact evaluations have provided evidence of accountability as 
well as change in R&D product users. 

This presentation was designed to illustrate an approach to 
evaluating the impact of vocational education R&D products by utilizing 
structural equation modeling. The objectives for the presentation 
were : 

(1) to describe the complexity of designing and conducting 
impact evaluation for R&D products, 

<2) to describe the R&D conceptual framework, 

<3) to apply LISREL to an R&D product impact model, and 

(4) to summarize the strengths and weaknesses of LISREL. 



There are several reasons why R&D product impact evaluations have 
been difficult to conduct. First, there are a number of types of R&D 
products. These R&D products range from information papers to 
curriculum guides to student handbooks. The varying features of these 
R&D products have made comparisons among R&D products difficult and 
clouded the expectations of evaluators. In addition, there are multiple 
users of R&D products. Some R&D products developed by The National 
Center for Research in Vocational Education (NCRVE) , for instance, have 
been used by such diverse types of organizations as international 
educational organizations, community colleges, local school boards and 
junior high schools. Certainly, the goals of these users differed. It 
was necessary to determine the individual user's goals in order to 
evaluate the effectiveness of R&D products. Furthermore, different 
stakeholders such as R&D product sponsors, administrators and developers 
held different expectations for product use. For example, R&D product 
sponsors have been concerned with the freqw*ncy of use or spread of 
various R&D products, while R&D product developers were more concerned 
with intensity of use within specific organizations. 

There were additional constraints placed upon R&D product 
evaluators. First, typically the time to conduct an evaluation was not 
limitless. A political mandate to determine the impact of R&D products 
sometimes prompted the onset of an impact evaluation too early. 
Furthermore, the cost of conducting R&D product evaluation has been 
high. The cost of collecting data has usually increased as methodology 
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has increasingly involved personal contact. Normally case studies have 
been more expensive to conduct than mail surveys. In order to ascertain 
individual goals of R&D product users, personal contact with R&D product 
users was necessary. Finally, some evaluators have advocated the use of 
naturalistic inquiry -for R&D product evaluations because of the 
diversity of R&D products, differing individual goals, and varying roles 
and organizations oi users. These evaluators have found that more 
rigorous approaches such as quasi-experimental or experimental designs 
have placed unrealistic controls on R&D product users. 

R&D Conceptual Framework 

Many evaluators of R&D products have searched for general 
guidelines to assist them in conducting practical, appropriate and 
useful R&D product impact evaluations. In order to address this 
problem, an internal evaluation team at NCRUE developed a conceptual 
framework for evaluating R&D product impact. The conceptual framework 
was developed to maximize the use of practical evaluation methods to 
address program improvement as well as impact. The evaluation team was 
committed to the following: 

• using multiple sources of data 

• accepting interwoven, and sometimes conflicting, 
program goals. 

• looking for accountability as well as effects data. 

The conceptual framework drew heavily on two bodies of literature 
<1) program evaluation and (2) change theory, especially the literature 
on diffusion of innovations. The conceptual framework was needed to 



help design impact evaluations in a cost effective and efficient manner. 
The primary goal of the conceptual framework was to present criteria 
which could be used to determine the likelihood of impact occurring 
based upon the degree to which quality, distribution, implementation and 
use of R&D products had occurred. Furthermore, the conceptual framework 
combined the goals of providing evidence to improve educational programs 
and document change. 

An evaluation team at NCRVE at The Ohio State University developed 
a conceptual framework to aid in estimating the impact of R&D products. 
This framework evolved over a five year period, 1978-82, as evaluations 
were conducted in a series of studies on vocational education R&D 
products. The central data base for constructing this framework was 
obt ned through impact studies of 28 selected state-developed and NCRVE 
products. Both qualitative and quantative data were collected through 
case studies and surveys. An extensive review of the diffusion 
literature revealed 267 impact studies of educational innovations. 

After the framework was developed, it was reviewed by participants 
in the fifth nationwide Vocational Education Dissemination and 
Utilization Conference, and revised. Since that conference, the 
framework and criteria have been reviewed by nine experts in 
dissemination and evaluation establishing its content validity. Three 
important assumpcions made in the development of the framework were: 

(1) New ideas could be packaged in a transportable format 
for use in diverse settings. 

<2) The primary output from a funded R&D project, normally a 
R&D product, was the most usable vehicle for tracing 



effects of the project. 

(3) Accountability was the driving -force behind most impact 
studies. 

The conceptual -framework -for measuring the impact of R&D products 
was a linear model with five stages; development, distribution, 
implementation, utilization and effects (Figure 1). These five stages 
depicted the logical flow of product development to program improvement. 
The importance of viewing program improvement as an interactive and 
cyclical process was represented by the dotted feedback line in the 
f i gure . 

A conceptual framework illustrating the five stages and criteria 
were provided through figure 2. The conceptual framework presented two 
different types of impact criteria} formative and summative. Criteria 
inherent in the first four stages of the conceptual framework were 
considered to be formative impact criteria. The primary purpose of 
evaluations conducted during these stages was to enhance the impact 
potential of the R&D product. Effects criteria were considered 
summative. The model hypothesized that an R&D product that was 
systemically developed, strategically disseminated, selectively 
implemented and used in an integrated manner was more likely to result 
in greater impact than a product which did not meet the** criteria 
(Hull, Adams and Bragg, 1983). 

Briefly, the stages were defined to clarify the intent of the 
conceptual framework. (The definitions of the R&D impact criteria were 
presented in Appendix A.) Development reitrred to the way the R&D 
product was produced. It was essential that R&D products be high in 
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quality 9 as evidenced through scholarship) to have impact. In addition, 
it was critical that the R&D product be relevant U the needs of the 
potential adoption site. 

D i str i but i on , the second stage, was clearly an important stage in 
the program improvement process. Distribution could occur through 
several means, including R&D product developer sponsored dissemination 
through targeted mailings. Distribution through a number of channels 
including dissemination of promotional materials, R&D product catalogs 
or conference presentations was necessary to obtain widespread 
acceptance. Typically, dissemination was the responsibility of the 
sponsoring agency sine* other users had -few incentives to distribute R&D 
products beyond their organization. However, secondary distribution was 
measured when it occurred. 

Imolementat ion bridged the gap between learning about the 
innovation 2nd trying it. The process of incorporating an innovation 
into an organization was very complex. Fullan and fomfret (1977), in 
their review oi curriculum implementation studies, identified the 
-following -factors influencing effective implementations 

<1) characteristics of the innovation, such as its 
compl ex i ty ; 

(2) strategies of implementation dealing with resource 
support, timing, and feedback mechanisms; 

(3) characteristics of the adopting units, such as their 
demographics and ability to solve problems; and 
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<4) political characteristics of the adopting unit such 
as incentive systems, the role o-f evaluation, and 
pol 1 1 ical compl ex i ty . 

Ut i 1 i zat i on of a R&D product must have occurred prior to evaluation 

o-f effects. An important concept analogous to utilization was sustained 

use. Causal models constructed by Crandall, Bauchner, Loucks and 

Schmidt (1982) showed that teacher commitment and elapsed time (i.e., 

the length of time the teacher has been using the innovation) were 

significant predictors o-f change in practice. There were several levels 

o-f use of an R&D product (Hall and Loucks, 1977). Only parts o-f a 

product may have been used or the product may have been used for only a 

short time period. Furthermore, a product may have been modified as it 

was used so that a -fairly new innovation had evolved. These levels o-f 

use must be identified to determine the extent that impact could be 

attributed to use of the original R&D product or tc modi-fied R&D 

products. 

The final stage of the conceptual framework was the effects stage. 
Measuring the effects of a R&D product constituted a search for vis.ble 
changes as a result of use. Impact could have occurred in subtle and 
individualized ways or occurred for society as a whole. Certainly, 
individualized changes have been more readily measured than changes in 
society. Impact upon society has been extremely difficult to measure 
due to the inability to limit the benefits of a publicly developed and 
distributed R&D product to only certain members of society. Thus, the 
benefits have typically been so diverse and widely dispersed that they 
have been difficult to identify. 
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The R&D Product Impact Model and LISREL 

Thus -far, the conceptual -framework has been developed based upon an 
extensive literature base, an impact evaluation data base, and the 
knowledge and expertise of an NCRVE evaluation team. In addition, the 
conceptual -framework has been reviewed by other experts to establish 
content validity. Yet, an important question still remains, "how could 
the R&D conceptual framework be used to measure the impact o-f vocational 
education R&D products?" 

The conceptual -framework represented a general model identifying 
the criteria necessary to measure R&D product impact in diverse settings 
and with different types of products, programs and people. Yet, the 
conceptual framework contained a cumbersome number of criteria to build 
into an R&D product impact evaluation model. In addition, the proposed 
relationships among criteria and the formative and summative stages were 
un tested . 

In order to evaluate the conceptual framework and establish its 
utility for R&D product impact evaluation, a more parsimonious model was 
designed. The criteria which were addressed in the model were three 
formative cr i ter i a wh i ch were user orientation, support systems and 
integrated utilization! and two summative criteria which were user 
satisfaction and individual growth (Figure 3). 

This model proposed that to the extent formative criteria had been 
met, there was increased likelihood users were satisfied with the. 
product and individual growth had occurred. The model hypothesized the 
following relationships among criteria: 
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(1) The R&D product which was user oriented influenced 

user satisfaction and individual growth. In addition, 
user oriented R&D products influenced integrated 
utilization which, in turn, influenced user satisfaction 
and individual growth. 

<2) Support systems influenced integrated utilization which, 
in turn, influenced user satisfaction and individual 
growth. Support systems did not directly influence 
user satisfaction and individual growth. 

(3) User satisfaction influenced individual growth and 
individual growth influenced user satisfaction. 

More specifically, the model hypothesized that the more R&D 
products were developed wi th the involvement of relevant user audiences 
through reviews and field testing, the more likely R&D products would be 
integrated into routine use within an organization and then influence 
the satisf ication of users and change in user's knowledge, skills and/or 
attitudes. Previous research has supported this premise as well as 
other aspects of the model. Hood and Blackwell (1976) and Seiber < 1981 ) 
found that products developed with input from local educators and 
products containing relevant information were more likely to be used. 
In addition, the more R&D products were user oriented, the more likely 
R&D products were to directly influence the satisfaction of users and 
directly influence individual growth. 

Furthermore, the more support systems in the form of personnel, 
information and financial resources available when R&D prooicts were 
implemented, the more likely R&D products would be used routinely within 
an organization and, in turn, influence the satisfaction of users and 
growth of knowledge, attitudes and/or skills of individuals. Support 
systems would not be expected to directly influence user satisfaction 
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and individual growth. Support resources were important -for 
implementation of a R&D product but not necessarily apparent when a R&D 
product was used. A R&D product user was satisfied or changed as a 
result o-f using the product, not by the support system. Berman and 
Pauly < 1 975) -found that without support personnel and physical 
resources, teachers have -frequently reported problems when implementing 
products. Berman and McLaughl i n < 1 978) indicated the importance o-f 
support personnel as change agents to improve implementation. 

Integrated utilization was the intensive and pervasive 
incorporation o-f R&D products into organizational routine. The more R&D 
products were integrated into use in an organization, the more R&D 
products would influence user satisfaction and individual growth. 
Crandall et al . < 1 982) described the importance of integrated 
utilization as evidenced by teacher commitment to a R&D product. Seiber 
< 1 981 ) described R&D product ownership as a powerful incentive for 
change . 

Finally, an individual that was satisfied with a R&D product was 
more likely to experience growth in knowledge, attitudes and/or skills 
than a dissatisfied product user. Furthermore, when individuals 
experienced growth, they were likely to attribute that change to use of 
the R&D product and be satisfied. 

A modeling approach, developed in recent years, known as 
structural equation modeling can be used to determine the degree to 
whijh the proposed model actually represented the R&D product impact 
phenomenon in the population. LISREL, an acronym for linear structural 



relationships, is a computer program developed by Joreskog during the 
1970's to analyze structural equation models <Joreskog and Sorbum, 
1983). L1SREL has been praised highly by researchers and statisticians 
in the social science -fields as a significant advancement which may 
enable researchers to study more complex phenomena than was previously 
possi bl e . 

LISREL is unique -from other modeling approaches, such as path 
analysis, since it enables analysis o-f causal structure among 
hypothetical constructs, also re-ferred to as latent variables. LISREL 
is based on maximum-likelihood statistical theory rather than the least 
squares statistical theory on which linear regression is based. 
However, like other modeling approaches, LISREL may be used to test 
causal linkages among variables by using data usually collected through 
non-experimental designs. The use o-f LISREL demands a priori theory 
regarding the relationships among variables. Based on theory, LISREL 
can test the causal linkages among latent and manifest variables. 
LISREL is predicated on a confirmatory approach and is de-finately not 
intended to be used in an exploratory manner! Without a priori theory, 
even though a model may have been tested and appeared tenable, a 
researcher can say little about the meaning-fulness o-f the relationships 
among var i abl es . 

The model shown in -figure 3 is re-ferred to as the structural 
equation model. The one-way arrows between two variables in the model 
indicated a postulated direct influence o-f one variable on another. 
Unlike path analysis, LISREL can handle a nonrecursive model illustrated 
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by the two arrows between user satisfaction and individual growth. This 
nonrecursive model implies user satisfaction directly influences 
individual growth and individual growth directly influences user 
sat i sf act i on . 

The structural equation model refers to relationships among 
exogenous and endogenous variables. The exogenous variables in this 
model were user orientation and support systems and the endogenous 
var i abl es were integrated ut i 1 i zat i on , user satisfaction and individual 
growth. Exogenous variables are independent variables, or can be 
thought of as causes, and assumed to be determined by causes outside the 
model. In contrast, endogenous variables are explained by other 
exogenous or endogenous variables. These endogenous variables can be 
thought of as effects, or dependent variables. 

As stated previously, LISREL has the capability of examining the 
relationships among latent variables, rather than only observed 
variables as occurs with path analysis. This second feature of LISREL 
allows the incorporation of the measurement model which specifies the 
relationships between latent and measured variables, also referred to as 
manifest variables. The measurement model is comparable to common 
factor analysis. Figure 4 illustrates the incorporation of both the 
structural equation model and the measurement model. Latent variables 
are traditionally shown in circles while manifest variables are placed 
in boxes. The error terms corresponding to each manifest and latent 
variable have been ommitted so the model is easier to visualize. 

Kerlinger (1979) pointed tc the need to utilize multiple indicators 
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to deal with reliability problems obtained during measurement. He 
continued by describing the complexity and indirect nature of many 
measurement procedures. Analytical approaches such as path analysis 
were limited since they did not possess the capability to incorporate 
multiple indicators or latent variables into the model. Most analytical 
approaches developed prior to LISREL assumed variables were measured 
without error. Pedhazer (1982) descr i bed measures in socal and behavior 
science as moderately reliable, at best. Furthermore, he explained that 
it was unrealistic to expect single indicators to capture the validity 
and reliability of the complex constructs typically studied in the 
social and behavioral sciences. 

Certainly, given the complexity of measuring R&D product impact, 
there was a need to carefully plan indicators -for each summative and 
■formative criteria. By incorporating the use of multiple data sources, 
the following indicators were identified for each formative criterion. 
Two i ndi cators were identified for user orientation wh i ch were _rel evance 
to the needs of the user and ease-of-use of the product. The four 
indicators of support systems were Administrative aupport, £hysical 
£esources, inservice training, and fj nanc i al support. Two indicators 
were identif ied f or i ntegrated ut i 1 i zat i on wh i ch were the pervasiveness 
and Intensity of use of the R&D product. The indicator of user 
satisfaction was an overall measure of satisfaction. Finally, based on 
the nature of the R&D product, a general assessment of the _knowl edge , 
skills or attitude was obtained. 

Several of the indicators would be measured through surveys of R&D 
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product users. (Appendix 6 contains examples of survey items -for the 
indicators o-f three latent variables.) For example, R&D product users 
could rate -from poor to excellent the quality o-f the product on 
relevance and ease o-f use. A second example would be to obtain an 
estimate o-f R&D product users about the amount o-f time devoted to 
inservice training during implementation o-f the R&D product. 

Once a theory has been developed, the structural equation model and 
measurement model have been speci-fied, and data have been collected, the 
researcher is ready to analyze data. The data are input -for analysis by 
LISREL in the -form o-f a correlation or covariance matrix among all tne 
mani-fest variables. In addition, the researcher speci-fies the 
relationships among variables -for the structural equation model and 
measurement model through eight matrices, rather than entering the raw 
data as would occur when using other social science statistical packages 
such as SAS or SPSS. A b^sic understanding o-f matrix algebra is 
necessary to use LISREL. However, once matrix algebra has been 
mastered, LISREL represents a -fairly simple analysis procedure to use. 

The results o-f a LISREL analysis will give the researcher evidence 
o-f the -fit of the model in the population and the -fit o-f individual 
parameters. Each element entered into the eight matrices, representing 
the relationships among variables in the population, is a parameter. 
The test statistic for the overall model and parameters is chi-square. 
Chi-square is an extremely power-ful test statistic -for LISREL models, 
partly because of its relationship to sample size. As the sample size 
increases, chi-square is more likely to be significant and thus, the 
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model is more likely to be rejected. So, since models are usually 
tested with large sample sizes, the models are usually rejected. 
Certainly, this is a sticky problem and researchers need to be aware o-f 
a number o-f other descriptive statistics, discussed by Bentler and 
Bonett (1980), to assess the -fit o-f the model. 

In summary, the R&D product impact model has illustrated the basic 
components o-f LISREL which were the structural equation model and the 
measurement model. Certainly, mani-fest war i abl es were needed to deal 
with the error o-f measurement o-f the hypothetical constructs in the R&D 
product impact model such as user orientation and integrated 
utilization. LISREL, based on max imum-1 i kel i hood stat i st i cal theory, 
would be used to test the -fit o-f the proposed model in the population, 
based on a sample o-f R&D product users. This paper has only touched 
upon the ways to use LISREL to test general models. For example, the 
R&D product impact model could be tested simultaneously with R&D product 
users representing di-f-ferent roles and organizations. However, caution 
has been made about the direct relationship between chi-square and 
sample size. A list o-f selected LISREL re-ferences has been provided -for 
those researchers who are interested in pursuing LISREL in the -future. 
In addition, the -final section o-f this paper highlights some o-f the 
strengths and weaknesses o-f LISREL not previously discussed. 

Strengths and weaknesses o-f LISREL 

LISREL possesses a number o-f -features which bring enormous 
potential -for analyzing other vocational education R&D phenomena. Some 
o-f the strengths o-f LISREL ares 
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<1) LISREL is a general modeling approach that can be 

used to do path analysis, multiple linear regression, 
common factor analysis as well as structural equation 
model ing. 

<2) LISREL represents a confirmatory approach <i.e. 

hypothesis testing) and thus, represents a rigorous 
approach to model testing. 

<3) LISREL allows the testing of general models to: 
<a) determine whether or not the models hold in the 
popul at i on , 

<b) compare the f i t of a general model across different 

populations, and 
<c) determine the -fit o-f competing general models. 

<4) LISREL forces the researcher to think through the 

theory under investigation including the relationships 
among variables. When using LISREL it is equally as 
important to specify missing paths as to specify 
existing paths. 

<5) LISREL can analyze models with correlated residuals 

(i.e. panel longitudinal data) *nd non-recursive models 
<i.e. user sa tisfaction and individual growth in the 
R&D product impact model). 

<6) LISREL can analyze categorical and continuous 

exogenous variables enabling the researcher to collect 
data through qualitative and quantitative methods. 

<7) General models can be modified, and the modifications 
tested, when the changes in the model are consistent 
with the a priori theory- and are conducted in a 
systemat i c manner . 

Some of the weaknesses of LISREL are the following: 



<1) LISREL requires a very large sample size, however 
a large sample size virtually guarantees that the 
model will be rejected. 

(2) LISREL can not analyze correlated relationships among 

endogenous variables, however LISREL can analyze correlation 
among exogenous variables. This restriction, however, 
limits the theories that can be tested with LISREL. 
A general model being developed by Bentler and Weeks 
will have the capability to analyze an even greater 
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variety of relationships among variables than LISREL 
(MacCallum, 1984). 

<3) LISREL users must be very cautious when modifying models 

because modifications can easily bring about an exploratory 
rather than a confirmatory approach. When this occurs, 
inferential statistics can no longer be applied since 
the researcher may be cap til i z i n g on chance. The LISREL 
literature related to modification of models should be 
reviewed prior to attempting to modify a model. 

(4) LISREL is expensive because of computer time and 
data collection with multiple indicators and large 
sample sizes. 

<5) LISREL is not as user-friendly as other social 

science statistical packages such as SAS and SPSS . 
Error messages are not as explicit as those provided by 
SAS or SPSS. It is easy to unknowingly specify the 
elements in the eight matrices incorrectly and thus, 
test a different model than the researcher had intended. 
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APPENDIX A 



The Criteria Defined 
Systematic development . A systematic process should be followed in devel- 
oping R&D products. An Ideal process would Include conducting research and 
needs assessment/ task analysis; reviewing relevant knowledge and practice; 
building a conceptual framework; sequencing development; conducting testing 
and revision cycles; disseminating the product; Implementing the product; and 
evaluating the results. 

High quality . Products should reflect .scholarship, be useful, communicate 
clearly, be marketable, and be free ol biases. Content should be accurate, 
up-to-date, focused on essentials, and complete. 

User orientation . Representatives of relevant audiences should be Identi- 
fied and Involved In designing, testing, and using Innovations. Primary audi- 
ences should receive priority In dissemination efforts. The resulting product 
should contain practical Information organized In an easy-to-use format. 

Strategic distribution . Cost-effective strategies for dietributiug an R&D 
product should be devised on the basis of the characteristics of potential 
users, site-specific factors, and features of the product Itself. Distribu- 
tion should reach oplrlon leaders and Influential organizations In the exter- 
nal environment. 

Multiple channels . More than one channel for conveying Information about 
products should be used. Communication should Include mass media (e.g., bro- 
chures sent out by mall) and Interpersonal channels (e.g., technical assis- 
tance). Normally,' In format Ion duplication and overlap are assets rather than 
liabilities during the distribution stage. 

Widespread distribution . Products should raach appropriate users. Thus, 
distribution to individuals in different roles, in diverse settings, and In 
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■any geographic areas aay be necessary. Secondary distribution through work- 
shops, reprints, libraries, the ERIC system, and so on should be encouraged* 

Sequential implementation . The introduction of products should be 
sequenced to meet the needs and unique characteristics of an adopting site. 
Often potential users need to be introduced to segments of the product to 
avoid total rejection of the Intervention. 

Support systems . Support systems necessary for encouraging the full use 
of a product should be operstional at the time of implementation. These sys- 
tems are of three types.' personal resources (e.g., administrative endorsement 
or site personnel endorsement), information resources (e»g.» training in the 
use of support materials and procedures), and physical resources (e.g., dol- 
lars, supplies, and equipment). 

Cost feasibility . Information describing the product's resource require- 
ments should allow quick and easy estimates of the costs likely to be Incurred 
by an adopting unit. 

Multiple patterns of use . A product's use patterns will vary according to 
the conditions, intensity level, frequency, and extent of use. The users' 
setting, role, and demographic characteristics create the conditions for dif- 

a 

ferent types of use* Multiple patterns of use and secondary use of R&D by 
other than the primary user audience should be encouraged. 

Time on task s An R&D product should be used frequently enough and long 
enough for its use to become an integral part of current practice. The audi- 
ence's time in actually using the product should be maximized* 

Integrated use * Use of a product should be intensive *nd pervasive 
throughout the organisations To accomplish this goal, personal commitment is 
required within the organization at all levels to institutionalize the product 
into organisational routines* 
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User satisfaction . The R&D product and its implementation should sett 

users 9 expectations and result In a positive user attitude toward the product. 
User satisfaction may be indicated by product advocacy or by creative 
adaptations. 

Individual growth . Products should contribute to changes in an individ- 
ual's attitude, knowledge, or performance. 

Organizational change . R&D products should contribute to beneficial 
changes in the user 9 6 organisational policy, programs, practices, or struc- 
ture. Beneficial changes may also include cost and time savings over current 
practice . 

Societal contributions . R&D products should contribute new and signifi- 
cant information with the potential to advance knowledge, improve current 
practice, or influence social systems. 
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APPENDIX B* 

User Orientation 

How would you rate the duality of the innovation on the 
following criteria? (One rating per criteria.) 

Not Very 
Criteria Applicable Poor Fair roo^ Good Excellent 

Relevance to my 

needs 0 1 2 3 4 5 

Ease of use 0 1 2 3 4 5 

Support Systems 

To what extent were adequate support systems available for 
implementing the innovation? 



ADEQUACY 



ERIC 





Not 


To Some 




To a 


TYPF OF SUPPOFT 


at All 


Fytent 




Great Fxter.t 


Administrative endorsement 


1 2 


3 


4 


F 


Personnel involved 


1 2 


3 


4 




Support material 


1 2 


3 


4 


«; 


Training 


1 2 


3 


4 


5 


Funds 


1 2 


3 


4 


5 


Equipment 


1 2 


3 


4 


5 


Supplies 


1 2 


3 


4 


5 



Pervasiveness/ Integrated Utilization 

Record the amount of classroom time (in person-days, P hours 
■ 1 day) you spent actually usino different sections of the 
innovation during the past year. 





Jan. 1 - 
March 31 


April 1 - 
June 30 


July 1 - 
Sept. 30 


Oct. 1 - 
Tec. 31 


TOTAL 
PFFSON-PAYS 


Section I 
Section II 
• 

Section N 













0 * See Hull, Adams and Bragg (1983) for additional eurvey Items. COPY 
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